The present invention relates to a method for computing encoding values of object points of three-dimensional scenes which include transparent objects. It further relates to a computing unit in which this method is implemented.
The method can be applied for the computation of computer-generated holograms for holographic display systems or for the production of holograms (hard copy). It can further be used with other three-dimensional display systems where object points can be displayed separately in a staggered manner in space, such as volumetric displays.
Generally, the present invention can also be used for other wavelength ranges than the visible spectral range. In conjunction with an antenna array where always at least two antennas emit coherent radiation so that the emitted electromagnetic waves can interfere with each other, it can be used for simulating and reconstructing electromagnetic spectra, for example in the context of spatial analysis of cosmic radiation received by radio telescopes. The spectral range which is used for such simulation or reconstruction does not necessarily have to correspond with the spectral range which is to be analysed, but can be imaged onto the former by way of transformation.
The present invention can further be applied to other media than the electromagnetic spectrum, e.g. to sound waves. In conjunction with an array of sound generating means where always at least two sound generating means can be controlled to emit coherent waves so that the emitted sound waves can interfere with each other, it can be used for simulating and reconstructing three-dimensional sound fields, where this invention shall not be limited to the audible sound frequency range. The sound fields comprise spatial and temporal varying sound values of three-dimensional scenes which include objects with sound-absorbing properties. The method and computing device can also be used to generate antiphase sound for reducing noise not only on a small place but also in a large environment.
The method can also be used for display and analysis of other spatial distributions, which can also be of non-optical nature. Three-dimensional distributions of physical and other parameters are imaged to transparency values, three-dimensional objects and light sources (false-colour imaging). It is for example preferably possible to visualise or analyse various tomographic methods, 3D ultrasonic checks or the distribution of mechanical stress in workpieces.
A holographic display system (in the following also simply denoted as a holographic system) according to this patent application is a display device for three-dimensional object data where the three-dimensional object data of the scene to be represented are encoded in the form of diffraction patterns of the scene to be reconstructed. Especially the computation of the diffraction patterns will be referred herein as encoding, and a number of encoding methods as such are already known.
The encoding can be achieved by generating aggregate holograms of the information of all object points, which can, however, easily cause a great computational load in particular with high-resolution display systems.
According to a further method, the hologram is divided into individual adjoining regions of same size (hogels) in order to minimise the computational load. Each region thus corresponds with an identical number of cells of the spatial light modulator (SLM) used. Each hogel carries information on a number of object points and on a multitude of diffraction angles (hogel vectors). The simplification is achieved in that pre-computed diffraction patterns can be retrieved from a look-up table (LUT) when computing the hogels.
Alternatively, the computation can be carried out separately for individual object points in the form of sub-holograms. Each sub-hologram is only written to a sub-region of the modulator surface of the optical light modulator (or spatial light modulator, SLM) which is used for the reconstruction. The individual sub-holograms can partly or wholly overlap on the modulator surface, depending on the position of the object points. This method can particularly preferably be applied if the hologram shall only be encoded for a small visibility region, where a at least one means is provided for tracking one or multiple visibility regions which are assigned to an observer eye to the movements of observer eyes of one or multiple observers. Such a holographic display device has for example been described by the applicant in document DE 103 53 439 B4 and in document WO 2006/066919 A1. The sub-holograms correspond with diffraction lenses which focus the desired object point with the desired brightness or with the desired brightness and colour at the desired distance to the modulator surface. The function of a convex lens is used to generate an object point in front of the modulator surface. The function of a concave lens is used to generate a virtual object point behind the modulator surface. An object point which lies in the modulator surface is generated directly. The lens functions can again be pre-computed and stored in a look-up table. When encoding the diffraction patterns, additional parameters, which can e.g. take into account the transfer functions of the used modulator regions of the SLM, light sources and other optical components in the optical path, can be considered. This also includes techniques which aim to reduce speckle.
Since in most displays individual pixels are represented on a planar SLM surface, a pixelated 2D image or a stereoscopic 3D representation which comprises at least two different 2D images (3D display) can be shown directly on those displays without much adaptation efforts needed. Necessary adaptations relate mainly to scaling the region to be represented to the resolution of the display panel and to brightness and colour adaptations to the gradation of the display panel. In a 3D display, multiple views of a stereoscopic representation must be encoded temporally and/or spatially on the modulator surface, depending on the used method. 2D vector graphics images must be transformed into raster graphics images before they can be displayed.
Before a three-dimensional scene can be represented on a 2D display or on a 3D display, or before it can be encoded for reconstruction in a holographic display, views must be generated from the three-dimensional data records which describe the objects of the scene with their properties. This process is also referred to as image synthesis or rendering. A number of methods are known for this which differ in the kind of scene description, the desired quality of the views and the way these views are actually generated.
For example, a 3D CAD model comprises geometric descriptions of the objects it includes in a three-dimensional coordinate system. In addition, a number of further physical properties can be defined to describe the materials of the objects, including optical properties such as reflectivity and emissivity of opaque objects and, additionally, refractive index and absorptivity of transparent objects. With homogeneous objects, it is sufficient that these parameters are defined for the boundary surfaces only. Generally, these properties can show not only a spatial gradient, and they can depend on one or multiple other parameters, such as wavelength and polarisation.
The data can also already exist in the form of volumetric pixel data. This is often the case with medical applications, for example. The 3D scene is divided into individual spatial points or small spatial regions (voxels) already when it is generated.
It is for example also possible that a 3D scene is generated from pixelated 2D data in combination with a depth map. The distance of each pixel to a reference plane is stored in the depth map. Such a data format is for example used for video data which shall be represented both on a 2D monitor and, additionally, on various 3D display devices. It facilitates the generation of multiple views of one scene. However, additional data must be provided to be able to consider hidden objects.
At the beginning of the image synthesis, a position must be chosen for each view to be generated in the three-dimensional coordinate system which serves to describe the location of objects in the scene, said position corresponding with the position of a camera with which a view of the scene could be recorded (virtual camera). Further, the virtual position and virtual size in the scene of the active modulator surface of the SLM which is used for image generation must be defined. The virtual size of the active modulator surface can differ from its actual size, e.g. if a scanning arrangement or a projection arrangement is used. The position of the virtual camera defines the position from which and the direction in which an observer eye would perceive the scene. This position can also lie between objects or in an object. The properties of the virtual camera such as focal length and viewing angle determine which section is displayed at which virtual magnification. The viewing angle is determined by the virtual area of the SLM and its position in relation to the virtual camera. The beams which originate in the position of the virtual camera and run through the borders of the virtual area of the SLM define a space which represents the visibility region. Parts of the scene which lie outside this pyramid cannot be displayed. In a 2D display the same view is generated for either observer eye, so that only perspective views are possible. By moving the virtual cameras for either observer eye in synchronism, the observer can virtually move through a scene during an image sequence while the observer does not have to move physically in front of the display. If the movement of the observer eyes in front of the display is detected by a sensor, the movement of the virtual camera can also be controlled based on this information. Further imaging means can be disposed between the virtual modulator surface and the observer eye. These imaging means can be included in the area of the virtual modulator surface and/or considered in the properties of the virtual camera.
In a holographic display, true depth information can be generated with the help of diffraction patterns. This gives an observer the possibility to focus at different depth planes of the reconstructed scene (accommodation) without the need to change the reconstruction. Therefore, it is rather referred to a virtual observer position than to a virtual camera in the context of a holographic display.
In the further course of image synthesis, it is determined which parts of the scene lie inside the visibility region and which parts are actually visible, e.g. which are not hidden behind other parts of the scene. This can be a multi-stage process, where the effort to be taken is the greater the more complex the scene or the more realistic the desired representation. Depending on the material properties and position of the light sources in the scene, it is possible to consider reflections, diffraction, refraction and scattering, which may in turn bring about further visible virtual objects, surfaces or points which are generated by parts of the scene which are visible, hidden and/or which lie outside the visibility region.
The appearance of the surfaces in the scene can be computed considering the material properties of the objects (shading). This includes for example an imaging of textures to the surfaces of the objects (texture mapping). Because the image synthesis is a very complex process, the appearance of objects, surfaces and individual image points can change several times during the image synthesis.
If the scene includes structured light sources, then their influence (illumination, shading) can be considered by adapting the appearance of surfaces, where often simplified illumination models are used in order to minimise the computational load. The reflectivity of the surfaces is often computed using bidirectional reflectance distribution functions (BRDF).
Recursive ray tracing methods are often used to generate the actual view of the scene. This means that the path of individual rays of light which are defined by a display pixel and the position of the virtual camera is traced back. First, all points at which the ray pierces through non-hidden surfaces of hit objects are determined and sorted by their distance to the virtual camera. Then, aggregate data is generated to describe a point of the view to be represented at the position of the corresponding display pixel, considering the appearance of all visible points involved. When generating this aggregated data, the transparency properties of all transparent points involved and, if there is one, of the opaque point are considered one after another. The transparency properties can be determined e.g. considering the material properties which determine the transparency and the optical path length which is covered by the ray of light in the material. Spectral and spatial distributions of these material properties can also be considered.
Such a method is described in U.S. Pat. No. 7,030,887 B2. Using multiple depth buffers in which depth information is stored, the transparent pixels which are mutually superposed are sorted by depth. This makes it possible to find the pixel which comes closest to an opaque pixel. Then, the transparency effect of this pixel in relation to the opaque pixel is computed. Then, it is found out whether or not there is an adjacent transparent pixel to the former transparent pixel. The transparency effect of this pixel is now computed in relation to the already computed transparency effect. This process is repeated until all superposed transparent pixels are considered. This method has the disadvantage that only one brightness value or one brightness value and one colour value is determined for each involved ray which corresponds to a pixel on the display panel.
There are similar problems when simulating sound fields in a room considering its acoustic and geometric properties (auralisation). Such simulations serve to circumvent extensive measurements in geometric models. Interrelations of sound sources of different location, movement, polar pattern and loudness and room acoustics can thus be tested. In addition to a having a certain position and form, individual objects in space or objects of the auditory scene also show wavelength-specific absorption and diffusion. The acoustic properties of a room are found in a multi-stage process, where recursive ray tracing methods are used as well. Again, it is possible to consider virtual sound sources, e.g. as caused by reflection, diffusion and deflection. The computed auditory sensation is typically rendered at the position of the virtual listener through stereo earphones, where the head-related transfer function (HRTF) must be considered for a realistic auditory sensation. It is a disadvantage here that it is only an aggregate signal that is rendered through the earphones. A round tour in a virtual room is nevertheless possible when re-computing the auditory sensation for the changed position of the listener, but a realistic auditory sensation without re-computing the sound signals after head movements is not possible.
The disadvantages when representing three-dimensional scenes which include transparent objects in a holographic display or volumetric display are overcome according to the present invention by the features of the method claimed in claim 1. As regards the field of acoustics, the disadvantages are overcome according to the present invention by the features of the method claimed in claim 8.